{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "ddf60620",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# print first n examples\n",
    "def print_first_n_items(dp, n = 1):\n",
    "    counter = 0\n",
    "    for d in dp:\n",
    "        if counter>=n:\n",
    "            break\n",
    "        print(d)\n",
    "        counter+=1\n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "839c377f",
   "metadata": {
    "scrolled": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('Architecturally, the school has a Catholic character. Atop the Main Building\\'s gold dome is a golden statue of the Virgin Mary. Immediately in front of the Main Building and facing it, is a copper statue of Christ with arms upraised with the legend \"Venite Ad Me Omnes\". Next to the Main Building is the Basilica of the Sacred Heart. Immediately behind the basilica is the Grotto, a Marian place of prayer and reflection. It is a replica of the grotto at Lourdes, France where the Virgin Mary reputedly appeared to Saint Bernadette Soubirous in 1858. At the end of the main drive (and in a direct line that connects through 3 statues and the Gold Dome), is a simple, modern stone statue of Mary.', 'To whom did the Virgin Mary allegedly appear in 1858 in Lourdes France?', ['Saint Bernadette Soubirous'], [515])\n",
      "('Architecturally, the school has a Catholic character. Atop the Main Building\\'s gold dome is a golden statue of the Virgin Mary. Immediately in front of the Main Building and facing it, is a copper statue of Christ with arms upraised with the legend \"Venite Ad Me Omnes\". Next to the Main Building is the Basilica of the Sacred Heart. Immediately behind the basilica is the Grotto, a Marian place of prayer and reflection. It is a replica of the grotto at Lourdes, France where the Virgin Mary reputedly appeared to Saint Bernadette Soubirous in 1858. At the end of the main drive (and in a direct line that connects through 3 statues and the Gold Dome), is a simple, modern stone statue of Mary.', 'What is in front of the Notre Dame Main Building?', ['a copper statue of Christ'], [188])\n",
      "('Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super Bowl title. The game was played on February 7, 2016, at Levi\\'s Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th Super Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of naming each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently feature the Arabic numerals 50.', 'Which NFL team represented the AFC at Super Bowl 50?', ['Denver Broncos', 'Denver Broncos', 'Denver Broncos'], [177, 177, 177])\n",
      "('Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super Bowl title. The game was played on February 7, 2016, at Levi\\'s Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th Super Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of naming each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently feature the Arabic numerals 50.', 'Which NFL team represented the NFC at Super Bowl 50?', ['Carolina Panthers', 'Carolina Panthers', 'Carolina Panthers'], [249, 249, 249])\n"
     ]
    }
   ],
   "source": [
    "from squad1 import SQuAD1\n",
    "\n",
    "# get train and dev dp using SQuAD1 factory function\n",
    "train_dp, dev_dp = SQuAD1(split=('train','dev'))\n",
    "# print first two training examples\n",
    "print_first_n_items(train_dp,2)\n",
    "\n",
    "# print first two dev examples\n",
    "print_first_n_items(dev_dp,2)\n",
    "                          "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "2bd4a0f8",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(3, \"Wall St. Bears Claw Back Into the Black (Reuters) Reuters - Short-sellers, Wall Street's dwindling\\\\band of ultra-cynics, are seeing green again.\"), (3, 'Carlyle Looks Toward Commercial Aerospace (Reuters) Reuters - Private investment firm Carlyle Group,\\\\which has a reputation for making well-timed and occasionally\\\\controversial plays in the defense industry, has quietly placed\\\\its bets on another part of the market.')]\n",
      "{'labels': [3, 3], 'text': [['Wall', 'St.', 'Bears', 'Claw', 'Back', 'Into', 'the', 'Black', '(Reuters)', 'Reuters', '-', 'Short-sellers,', 'Wall', \"Street's\", 'dwindling\\\\band', 'of', 'ultra-cynics,', 'are', 'seeing', 'green', 'again.'], ['Carlyle', 'Looks', 'Toward', 'Commercial', 'Aerospace', '(Reuters)', 'Reuters', '-', 'Private', 'investment', 'firm', 'Carlyle', 'Group,\\\\which', 'has', 'a', 'reputation', 'for', 'making', 'well-timed', 'and', 'occasionally\\\\controversial', 'plays', 'in', 'the', 'defense', 'industry,', 'has', 'quietly', 'placed\\\\its', 'bets', 'on', 'another', 'part', 'of', 'the', 'market.']]}\n"
     ]
    }
   ],
   "source": [
    "from ag_news import AG_NEWS\n",
    "\n",
    "# get train datapipe using AG_NEWS fatory function\n",
    "# Like previous example, use split = ('train','test') to get train and test datapipes\n",
    "train_dp = AG_NEWS(split='train')\n",
    "\n",
    "# create datapipe with batch size = 2\n",
    "train_dp = train_dp.batch(2)\n",
    "\n",
    "# print first batch with 2 examples\n",
    "print_first_n_items(train_dp)\n",
    "\n",
    "\n",
    "def _process_batch(batch):\n",
    "    return {'labels': [sample[0] for sample in batch], 'text': [sample[1].split() for sample in batch]}\n",
    "\n",
    "#Apply tokenization and create labels and text named batch\n",
    "train_dp = train_dp.map(_process_batch)\n",
    "print_first_n_items(train_dp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "b1401a57",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('you shoot something on crummy-looking videotape ', '0')\n",
      "('burdened by the actor ', '0')\n"
     ]
    }
   ],
   "source": [
    "from sst2 import SST2\n",
    "\n",
    "# get train datapipe using SST2 factory function\n",
    "train_dp = SST2(split='train')\n",
    "\n",
    "# shuffle dataset\n",
    "train_dp = train_dp.shuffle()\n",
    "\n",
    "# print first 2 random examples\n",
    "print_first_n_items(train_dp,2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "efe92627",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('neg', 'I rented I AM CURIOUS-YELLOW from my video store because of all the controversy that surrounded it when it was first released in 1967. I also heard that at first it was seized by U.S. customs if it ever tried to enter this country, therefore being a fan of films considered \"controversial\" I really had to see this for myself.<br /><br />The plot is centered around a young Swedish drama student named Lena who wants to learn everything she can about life. In particular she wants to focus her attentions to making some sort of documentary on what the average Swede thought about certain political issues such as the Vietnam War and race issues in the United States. In between asking politicians and ordinary denizens of Stockholm about their opinions on politics, she has sex with her drama teacher, classmates, and married men.<br /><br />What kills me about I AM CURIOUS-YELLOW is that 40 years ago, this was considered pornographic. Really, the sex and nudity scenes are few and far between, even then it\\'s not shot like some cheaply made porno. While my countrymen mind find it shocking, in reality sex and nudity are a major staple in Swedish cinema. Even Ingmar Bergman, arguably their answer to good old boy John Ford, had sex scenes in his films.<br /><br />I do commend the filmmakers for the fact that any sex shown in the film is shown for artistic purposes rather than just to shock people and make money to be shown in pornographic theaters in America. I AM CURIOUS-YELLOW is a good film for anyone wanting to study the meat and potatoes (no pun intended) of Swedish cinema. But really, this film doesn\\'t have much of a plot.')\n",
      "(0, 'I rented I AM CURIOUS-YELLOW from my video store because of all the controversy that surrounded it when it was first released in 1967. I also heard that at first it was seized by U.S. customs if it ever tried to enter this country, therefore being a fan of films considered \"controversial\" I really had to see this for myself.<br /><br />The plot is centered around a young Swedish drama student named Lena who wants to learn everything she can about life. In particular she wants to focus her attentions to making some sort of documentary on what the average Swede thought about certain political issues such as the Vietnam War and race issues in the United States. In between asking politicians and ordinary denizens of Stockholm about their opinions on politics, she has sex with her drama teacher, classmates, and married men.<br /><br />What kills me about I AM CURIOUS-YELLOW is that 40 years ago, this was considered pornographic. Really, the sex and nudity scenes are few and far between, even then it\\'s not shot like some cheaply made porno. While my countrymen mind find it shocking, in reality sex and nudity are a major staple in Swedish cinema. Even Ingmar Bergman, arguably their answer to good old boy John Ford, had sex scenes in his films.<br /><br />I do commend the filmmakers for the fact that any sex shown in the film is shown for artistic purposes rather than just to shock people and make money to be shown in pornographic theaters in America. I AM CURIOUS-YELLOW is a good film for anyone wanting to study the meat and potatoes (no pun intended) of Swedish cinema. But really, this film doesn\\'t have much of a plot.')\n"
     ]
    }
   ],
   "source": [
    "from imdb import IMDB\n",
    "\n",
    "train_dp = IMDB(split='train')\n",
    "print_first_n_items(train_dp)\n",
    "\n",
    "\n",
    "#convert label into integer using map\n",
    "labels = {'neg':0,'pos':1}\n",
    "\n",
    "def _process_tuple(x):\n",
    "    return labels[x[0]], x[1]\n",
    "\n",
    "train_dp = train_dp.map(_process_tuple)\n",
    "print_first_n_items(train_dp)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
